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Executive  Summary 


This  report  illustrates  the  concept  and  value  of  the  Risk  Diagnostic  Roadmap  (RDR),  a 
compendium  of  risk  identification  and  analysis  (RI&A)  techniques  for  programs  that  acquire 
and/or  develop  software  intensive  systems.  The  RDR  is  conceived  as  a  comprehensive 
reference  tool  that  not  only  would  provide  insight  into  various  risk  identification  and  analysis 
techniques,  but  also  would  enable  users  to  easily  compare  and  contrast  these  techniques.  As 
such,  it  would  be  a  powerful  resource  for  anyone  who  needs  to  assess  the  health  of  programs 
in  the  Department  of  Defense,  other  government  organizations,  or  in  private  industry.  This 
audience  includes  program  executive  officers,  acquisition  program  managers  (PMs),  program 
staff,  the  chief  engineers  at  the  Carnegie  Mellon®  Software  Engineering  Institute,  as  well  as 
the  larger  risk  management  community. 

Using  the  Risk  Diagnostic  Roadmap  would  provide  immediate  benefits  by  optimizing  the 
process  of  choosing,  recommending,  and  employing  RI&A  techniques.  Currently,  choosing 
among  so  many  RI&A  techniques  can  be  challenging  for  anyone  trying  to  select  or  use  such 
diagnostic  methods.  This  is  true  even  for  experts  in  risk  management;  experts  whose 
extensive  experience  with  select  RI&A  methods  may  make  them  less  likely  to  investigate, 
recommend,  or  use  unfamiliar  techniques.  The  RDR  could  address  this  problem  and  advance 
risk  management  practice  by  opening  the  full  range  of  RI&A  techniques  up  to  the  entire  user 
community. 

Beyond  these  immediate  benefits,  framers  of  the  RDR  envision  two  exciting  advances  that 
could  occur  with  its  creation  and  maintenance.  First,  the  RDR  will  inevitably  highlight  areas 
where  existing  diagnostic  methods  do  not  fully  address  the  needs  and  emerging  trends.  Thus, 
the  Risk  Diagnostic  Roadmap  would  become  an  invaluable  resource  for  those  seeking  to 
address  those  needs  and  trends,  either  by  performing  research  or  by  strengthening 
implementation  processes. 

Second,  the  effort  to  organize  the  Risk  Diagnostic  Roadmap  will  reveal  commonalities 
among  the  diagnostic  methods  that  could  lead  to  a  way  of  translating  the  output  (“leave- 
behind  data”)  of  such  methods  into  a  standardized  format  such  as  risk  statements. 

Currently,  the  output  (“leave-behind  data”)  of  one  technique  cannot  be  compared  effectively 
with  the  outputs  produced  by  alternative  methods.  With  standardized  output,  PMs  as  well  as 
the  larger  risk  management  community  could  better  evaluate  competing  risk  diagnostics. 


®  Carnegie  Mellon,  Architecture  Tradeoff  Analysis  Method,  ATAM,  Capability  Maturity  Model, 
CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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This,  in  turn  would  improve  PMs’  ability  to  accurately  measure  the  status  of  their  programs, 
while  also  improving  senior  acquisition  executives’  ability  to  view  of  program  risks  service¬ 
wide  and  across  services. 

At  this  point,  it  should  be  re-emphasized  that  the  Risk  Diagnostic  Roadmap  is  not  a  new 
diagnostic  method  or  tool.  It  is  a  framework  for  comparing  the  benefits,  strengths,  and 
advantages  of  existing  methods.  It  is  also  a  means  of  identifying  areas  for  future  RI&A 
research. 

To  that  end,  staff  members  at  the  SEI  have  begun  discussing  the  basis  for  including  or 
excluding  existing  methods  in  the  RDR.  Moreover,  we  have  begun  preparing  the  roadmap 
using  three  risk-based  diagnostic  tools  developed  at  the  institute:  the  Architecture  Tradeoff 
Analysis  Method®  (ATAM®),  and  Commercial  Off-the-Shelf  (COTS)  Usage  Risk  Evaluation 
(CUREsm),  and  the  Software  Risk  Evaluation  (SRE).  The  RDR  team  invites  critical  comment 
and  seeks  collaborators  willing  to  help  develop  version  1. 


SM  CURE  and  SCAMPI  are  services  marks  of  Carnegie  Mellon  University. 
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Abstract 


This  technical  note  illustrates  the  concept  and  value  of  the  Risk  Diagnostic  Roadmap  (RDR), 
which  is  envisioned  to  be  a  comprehensive  reference  tool  for  risk  identification  and  analysis 
(RI&A)  techniques.  Program  Managers  (PMs)  responsible  for  developing  or  acquiring 
software-intensive  systems  typically  identify  risks  in  different  ways.  Some  PMs  and 
consultants  rely  on  ffee-form  brainstorming  or  volunteered  statements.  Others  select  risk 
diagnostic  methods  based  on  convenience  and  familiarity.  Both  approaches  are  focused  more 
on  the  experience  and  knowledge  of  the  PM  and/or  consultant  than  on  the  requirements  of  the 
program. 

Researchers  at  the  Carnegie  Mellon®  Software  Engineering  Institute  are  developing  an 
alternative  approach  in  the  form  of  the  RDR.  The  RDR  is  populated  with  an  “appropriate”  set 
of  risk  diagnostic  methods.  The  roadmap  enables  PMs  to  compare  risk  diagnostic  methods 
and  choose  the  best  method(s)  for  their  particular  situations. 

This  technical  note  describes  the  evolution  of  the  Risk  Diagnostic  Roadmap  and  presents  the 
attributes  that  qualify  risk  diagnostic  tools  as  “appropriate”  for  the  roadmap.  SEI  researchers 
then  use  these  attributes  to  select  three  candidate  risk  diagnostic  methodologies  for  inclusion 
in  the  RDR. 
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1  Introduction:  What  is  a  Roadmap? 


The  concept  of  a  roadmap  is  not  new,  but  it  has  seen  increased  use  in  recent  years.  The  idea 
of  capturing  the  current  understanding  of  an  industry  area  or  product,  providing  a 
retrospective  of  “how  we  got  here,”  and  forecasting  the  future  of  the  field  has  piqued  the 
interest  of  professionals  in  various  industries  associated  with  manufacturing,  science,  and 
technology.  Roadmaps  and  similar  documents  have  been  produced  in  the  semi-conductor 
industry,  the  cyber-infrastructure,  the  wireless  domain,  and  the  Internet  (semantic  Web)  to 
name  a  few. 

In  reviewing  the  possibilities  of  presenting  a  technology  roadmap,  two  specific  concepts 
emerge.  The  first  is  the  process  involved  in  capturing  a  current  understanding  of  an  industry 
area  or  product.  Doing  so  can  provide  a  retrospective  of  “how  we  got  here,”  a  forecast  of 
“where  we  are  going”  or  a  hybrid  of  both. 

The  other  is  an  image  of  a  literal  roadmap,  with  graphical  depictions  of  destinations,  rural 
routes,  highways,  toll  roads,  national  parks  and  points  of  interest.  Robert  Schaller,  professor 
of  in  the  Department  of  Business  Economics  and  Legal  Studies  at  the  College  of  Southern 
Maryland,  offers  one  definition  of  roadmaps  as  follows: 

Generically,  a  "road  map"  is  a  layout  of  paths  or  routes  that  exist  (or  could 
exist)  in  some  particular  geographical  space.  In  everyday  life,  road  maps  are 
used  by  travelers  to  decide  among  alternative  routes  toward  a  physical 
destination.  Thus,  a  road  map  serves  as  a  traveler’s  tool  that  provides 
essential  understanding,  proximity,  direction,  and  some  degree  of  certainty  in 
travel  planning.1 

The  Risk  Diagnostic  Roadmap  is  envisioned  to  be  just  such  a  representation  with  “cities” 
categorized  as  entry  points  and  “destinations”  as  exit  criteria.  Traveling  between  the  cities 
are  the  various  routes  prescribed  by  the  PM  or  general  practitioner  that  will  be  the  most 
appropriate  for  the  RI&A  needs  of  the  organization.  After  reviewing  the  RDR,  programs  can 
choose  the  direction  among  the  various  possible  routes.  They  will  be  able  to  select  a  route 
that  will  take  them  though  a  series  of  diagnostics  to  get  them  to  their  final  destination  of 
improved  health. 


1  Schaller,  Robert  R.  2004.  Technological  Innovation  in  the  Semiconductor  Industry:  A  Case  Study  of 
the  International  Technology  Roadmap  for  Semiconductors  (ITRS),  PhD  Dissertation,  George 
Mason  University. 
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In  any  case,  the  creation  of  a  roadmap  requires  review  of  current  technology  and  technology 
trends,  as  well  as  real  and  envisioned  business  needs.  Doing  so  provides  a  context  for  future 
technological  development  in  a  particular  field.  The  Risk  Focus  Team  has  opted  to  focus 
effort  on  producing  the  broad  context  in  exploring  and  advancing  RI&A  application. 

Based  on  the  combined  experiences  of  the  Risk  Focus  Team,  the  following  initial  five  exit 
criteria  or  recommended  destinations  were  identified: 

Come  Back  Next  Year.  If  the  application  of  one  diagnostic  yields  results  that  indicate  no 
further  diagnostics  are  needed,  the  program  can  “come  back  next  year.” 

Lifestyle  Changes  Needed.  If  results  show  the  program  isn’t  in  immediate  danger,  but  could 
be  on  the  road  to  future  problems  then  “lifestyle  changes  are  needed.” 

Extreme  Measures  Needed.  If  results  of  a  diagnostic  show  that  the  program  may  be  in 
trouble,  but  urgent  action  can  save  it,  “extreme  measures  are  needed.” 

Out  of  Immediate  Danger.  If  one  or  a  series  of  RI&A  techniques  rescues  the  program,  it  is 
then  “out  of  immediate  danger,”  but  should  probably  continue  to  monitor  health  through  the 
general  practitioner  or  a  chief  engineer. 

Terminal  Condition.  In  the  most  unfortunate  circumstances,  if  results  provide  the  final 
evidence  that  recovery  is  not  possible;  a  “terminal  condition”  is  present.  The  program  should 
be  terminated. 

These  “exit  points”  are  analogous  to  the  kinds  of  outcomes  patients  would  expect  after  a 
period  of  exploratory  medical  diagnostics.  With  the  inclusion  of  other  RI&A  techniques  into 
the  future  Risk  Diagnostic  Roadmap,  other  valid  end  points  may  also  be  identified. 

To  fill  in  the  terrain  between  entry  and  exit  points,  the  three  risk-based  diagnostics  selected 
for  initial  analysis  will  be  used  to  depict  realistic  criteria  for  visiting  one  or  more  of  these 
destinations.  All  diagnostics  that  find  a  place  on  the  roadmap  will  also  be  of  value  to 
programs  at  some  point  in  their  life  cycle.  It  is  likely  that  once  the  first  RDR  is  created,  gaps 
will  be  identified  where  there  will  be  an  obvious  area  of  need  that  lacks  a  diagnostic.  In 
some  cases,  the  diagnostic  will  exist  and  will  be  added;  in  other  cases,  a  known  diagnostic 
will  not  exist,  and  framers  of  the  roadmap  will  recommend  development  to  fill  the  hole. 

1.1  Why  a  Risk  Diagnostic  Roadmap? 

As  in  the  examples  provided  above,  the  Risk  Diagnostic  Roadmap  will  allow  existing  RI&A 
techniques,  current  program  needs,  and  technology  trends  to  converge  in  a  coherent  and 
logical  framework. 
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Practitioners  and  consumers  of  these  diagnostics  will  have  a  clear  understanding  of  the  range 
of  possibilities  in  applying  specific  diagnostics  to  meet  their  particular  program  need.  For 
PMs  outside  this  area  of  expertise,  the  RDR  will  help  define  the  value  of  existing  techniques 
and  the  course  of  exploration  that  may  benefit  their  programs.  For  developers  of  diagnostics, 
the  RDR  will  either  validate  their  chosen  technical  direction  or  will  redirect  their  efforts  to 
ensure  more  relevance  and  applicability.  In  this  way,  the  RDR  will  help  shape  future  research 
and  development  to  meet  emerging  needs,  thereby  advancing  the  field  of  risk. 

As  part  of  this  effort,  the  Risk  Diagnostic  Roadmap  will  promote  the  use  of  a  standard  format 
for  representing  risk  items  all  RI&A  methods.  Once  in  the  standard  format,  the  risk  items  can 
then  be  entered  into  a  program’s  risk  repository.  All  program  risks,  whether  derived  from  a 
functional  architecture  or  elicited  from  program  staff  can  be  identified,  assigned  an  impact 
and  probability,  and  monitored,  not  uniquely,  but  systemically. 


1 .2  Why  the  Carnegie  Mellon  Software  Engineering  Institute 
(SEI)? 

Generally,  SEI  work  could  be  described  as  sharing  the  core  element  of  finding  ways  to 
identify  and  mitigate  risk.  SEI  work  has  either  helped  members  of  the  software  community 
discover  and  understand  the  risks  they  run,  or  given  them  specific  approaches  for  dealing 
with  those  risks.  For  example,  process  management  and  improvement  is  a  risk  mitigation 
strategy  for  a  whole  constellation  of  risks  that  may  not  be  explicitly  articulated.  Our 
commercial  off-the-shelf  software  (COTS)  and  product  line  efforts  have  yielded  strategies  to 
deal  with  risks  that  researchers  have  perceived  in  specific  technical  areas.  The  network 
systems  survivability  work  has  been  to  identify  risks  in  information  systems  and  mitigate 
them  as  soon  and  completely  as  possible.  Therefore,  developing  the  RDR  can  be  viewed  as  a 
natural  extension  of  our  efforts  to  further  the  state  of  the  practice. 
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2  Improving  Risk  Management  Practice  through  the 
RDR:  An  Analogy  between  Risk  Management  and 
Medical  Care 

Although  the  state  of  health  care  is  an  oft-debated  issue,  many  of  us  take  for  granted  that  a 
certain  level  of  sophistication— professional  licensure,  organizational  accreditation,  medical 
research,  treatment  protocols,  standards  of  care,  procedural  checks,  and  so  on — is  inherent  in 
the  health  care  system.  Examining  risk  identification  and  analysis  from  the  perspective  of 
medical  care  might  point  the  way  for  acquisition  to  take  a  few  steps  toward  an  analogous  care 
system. 

At  some  point  everyone  has  sought  help  from  doctors  and  hospitals  for  illness  or  injury,  so 
the  world  of  medical  care  is  a  familiar  one.  People  seeking  health  care  are  not  expected  to 
diagnose  and  treat  themselves;  instead  they  seek  the  help  of  medical  professionals.  Likewise, 
people  seeking  to  improve  the  health  of  their  programs  should  be  able  to  draw  upon  the 
expertise  of  a  community  of  risk  professionals. 

To  take  the  analogy  further,  consider  acquisition  support  as  if  it  were  “medical  care  for 
programs.”  Managers  who  wanted  to  check  the  health  of  their  programs  could  consult  an 
easy-to-use  self-diagnostic  tool — a  “thermometer.” 

If  the  thermometer  indicated  a  problem,  the  PM  could  then  schedule  a  “checkup” — consult  an 
RI&A  expert— to  identify  any  potential  risk  areas  for  preemptive  treatment  and/or  to  address 
small  problems  before  they  become  big  ones. 

Programs  with  clear,  urgent,  mission-threatening  problems  could  seek  “emergency  room” 
care — immediate,  expert  triage  and  intervention  to  preserve  the  life  of  the  program.  In  any  of 
these  scenarios,  programmers  could  be  called  “specialists”— experts  in  treating  particular 
risks  or  dysfunctions.  The  following  sections  describe  these  three  interventions  in  greater 
detail. 

2.1  The  Thermometer 

The  “thermometer”  is  conceived  as  a  self-diagnostic  tool  for  PMs  or  other  personnel.  The 
thermometer  would  provide  a  snapshot  measuring  current  risk  indicators  based  on  a  generally 
accepted  picture  of  program  health.  Users  would  have  the  option  to  mitigate  risks  on  their 
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own  or  seek  the  help  of  an  expert.  Developing  such  a  “thermometer”  could  represent  a 
project  for  near-term  research. 


2.2  The  Checkup 

If  necessary,  programs  consult  an  expert  analogous  to  a  general  practitioner  for  a  regularly 
scheduled  “checkup.”  The  expert  applies  simple  diagnostics— perhaps  in  the  form  of  surveys 
or  checklists.  Depending  on  the  results,  the  expert  might  prescribe  more  costly  diagnostics  or 
additional  modes  of  action.  As  with  the  thermometer,  PMs  could  use  the  expert  (“check-up” 
intervention)  on  a  regular  basis  to  track  program  risks  over  time. 


2.3  The  Emergency  Room 

“Emergency  room”  diagnostics  are  implemented  for  programs  requiring  immediate  attention 
(critical  care).  If  serious  problems  threaten  the  life  of  the  program,  the  PM  can  call  in 
experts— perhaps  even  a  team  of  specialists— to  diagnose  the  immediate/critical  issues  and 
determine  the  appropriate  “treatments.” 

For  the  “health  care”  model  of  acquisition  support  to  work,  all  diagnostic  techniques  should 
have  their  entry  points  expressed  as  one  of  these  modes  of  intervention—  “thermometer,” 
“checkup,”  or  “emergency  room.”  The  diagnostics  should  also  identify  exit  points  that  either 
would  indicate  the  need  for  further  testing  or  suggest  the  use  of  other  diagnostic  methods.  In 
the  medical  field,  these  results  and  recommendations  could  come  in  the  form  of  treatments, 
referrals  to  other  specialists,  etc.  In  the  RI&A  domain  they  might  include  tiger  teams, 
independent  technical  assessments,  and  so  on. 

This  structure  would  enable  PMs  to  take  advantage  of  the  range  of  RI&A  techniques. 
However  before  this  model  could  be  implemented,  RI&A  practitioners  need  to  know  what 
techniques  are  available,  when  and  in  what  situations  they  are  best  applied,  and  how  to  apply 
them  effectively  in  combination  or  in  sequence.  In  short,  they  need  the  roadmap. 
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3  Organizing  the  RDR 


SEI  technical  staff  members  initiated  the  road  mapping  activity  by  reviewing  a  discrete  set  of 
diagnostic  techniques.  To  be  included,  each  diagnostic  had  to  satisfy  three  criteria.  First,  it 
had  to  be  risk  based  (as  opposed  to  model  based,  see  the  following  section  for 
differentiation).  Second,  it  must  include  a  method  to  elicit  risk  items  (risk  statements).  And 
third,  it  must  be  familiar  to  those  staffing  this  project.  These  three  points  allowed  us  to  test 
assumptions  and  to  gain  a  better  understanding  of  the  diagnostics  selected  for  review. 


3.1  Risk  Based  and  Model  Based  Diagonistics 

Risk  based  Diagnostic 


Evaluated,  grouped, 
and/or  prioritized  risk#^ 


(optional) 


Recommendation 
and/or  Planning 
Phases 


Evaluation, 

categorization,  and . 

prioritization  information 


Per  CMMf 


Figure  1:  Risk  Diagnostics  Roadmap 

Diagnostics  appear  to  fall  into  two  broad  categories:  model  based  diagnostics  and  risk-based 
diagnostics.  Model  based  diagnostics  compare  real  programs  to  a  model  and  present  areas  of 
deviation  in  the  form  of  findings.  Standard  CMMI  Appraisal  Method  for  Process 
Improvement  (SCAMPIsm)  [Hays  93],  Capability  Maturity  Model®  (CMM®)-Based 
Appraisals  for  Internal  Process  Improvement  (CBA-IPI)  [Dunaway  01],  and  Software 
Capability  Evaluations  (SCE)  are  such  diagnostics.  Risk-based  diagnostics  focus  on  a 
program’s  current  practices  and  identify  risks  associated  with  them.  The  Software  Risk 
Evaluation  (SRE),  Architecture  Tradeoff  Analysis  Method®  (ATAM®)  [Kazman  00],  and  the 
Commercial  Off-the-Shelf  (COTS)  Usage  Risk  Evaluation  (CURESM)  [Alberts  01], 
exemplify  risk-based  diagnostics. 
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Returning  to  the  medical  analogy,  model-based  diagnostics  can  be  compared  to  certain  blood 
tests  like  lipids  tests,  in  which  subjects  are  compared  to  a  historically  determined  profile  for  a 
healthy  individual.  If  individuals  deviate  from  that  profile,  they  face  specific  risks,  e.g., 
increased  risk  of  heart  attack.  Risk-based  diagnostics  can  be  compared  to  magnetic  resolution 
images  (MRIs)  or  X-rays,  in  which  specialists  look  for  causes  of  a  patient’s  reported 
ailments. 

Because  the  risk-based  diagnostics  identified  above  were  developed  in  relative  isolation,  they 
can  be  more  confusing  to  novice  users.  For  that  reason,  they  were  appropriate  for  initial 
consideration  in  the  roadmap.  However,  model-based  diagnostics  certainly  have  a  place  in 
the  roadmap,  and  will  be  added  once  the  initial  structure  has  been  developed  and  refined. 


3.2  The  Importance  of  Risk  Items 

With  the  advent  of  the  Capability  Maturity  Model  Integrated  (CMMI®)  in  the  software 
community,  programs  and  organizations  have  begun  instituting  continuous  process 
improvement  to  achieve  Level  3  in  the  CMMI  staged  representation  CMMI  [SEI 01,  SEI 02]. 
Level  3  contains  three  specific  goals  and  seven  specific  practices  in  the  Risk  Management 
Process  Area.  These  goals  and  practices  form  a  logical,  single-pass  process  for  continuously 
identifying,  analyzing,  and  mitigating  risks. 

The  existence  of  at  least  one  risk  repository  is  implicit.  Every  risk  identified  for  the  program 
or  its  contractors — regardless  of  the  method  used  for  identification — should  find  its  way  to 
the  risk  repository.  Otherwise,  there  would  be  little  point  in  having  such  a  repository. 

But  what  are  “risks?”  Risk  items  could  be  descriptions  of  potential  calamities  that  could 
occur  to  a  program,  like  being  late,  burning  too  much  money,  or  building  something  that  the 
customer  doesn’t  want.  Risks  could  also  be  “if-then”  constructs.  The  definition  and  format  of 
risk  items  is  open,  and  for  the  most  part,  it  is  simply  left  to  the  discretion  of  the  user. 

Differences  among  risk  items  aside,  whatever  focus  the  various  risk-based  diagnostics  have, 
RDR  participants,  including  researchers,  PMs,  and  other  interested  parties)  will  need  to  feed 
the  risk  repository,  and  they  need  to  do  it  in  a  consistent,  mutually  supportive  way. 

The  definition  and  format  of  risk  items  is  discussed  further  in  the  technical  note,  Risk-Based 
Diagnostics.2 


2  Williams,  R.;  Ambrose,  K.;  Merendino,  T:;  &  Bentrem,  L.  Risk  Based  Diagnostics,  (CMU/SEI- 
2004-TN-013).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  2004. 
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4  Selecting  Initial  Diagnostics  for  Inclusion  in  the 
Roadmap 


Maturing  the  arena  of  RI&A  is  about  defining  its  parts  and  determining  where  the  software 
community  needs  to  focus  more  research.  Therefore,  we  want  to  cast  as  large  a  net  as 
possible  for  risk-based  diagnostics.  In  our  journey  toward  understanding  the  RI&A 
landscape,  we  began  with  diagnostics  that  are  reasonably  familiar  to  technical  staff  at  the 
SEI.  We  have  discerned  three  different  patterns  among  these  diagnostics  in  terms  of  how  they 
go  about  eliciting  the  risk  items  and  what  source  they  tend  to  lean  on  most  heavily:  (1) 
mining  the  knowledge  of  the  people  in  the  program  and  structuring  that  knowledge  into  risk 
items;  (2)  bringing  outside  expertise  to  bear,  and  using  that  expertise  to  structure  the  risk 
items;  and  (3)  providing  an  “expert  system”— a  tool  that  can  be  applied  consistently, 
independent  of  the  ability  of  the  program’s  personnel  or  that  of  the  outside  team. 

The  SRE  is  an  example  of  the  first;  the  ATAM  is  an  example  of  the  second;  and  CURE  is  an 
example  of  the  third.  In  addition,  we  have  classed  the  Software  Quality  Assessment  Exercise 
(SQAE3),  Operationally  Critical  Threat,  Asset,  and  Vulnerability  Evaluation  (OCTAVE®), 
and  most  Independent  Technical  Assessments  (ITAs)  as  you  see  here. 


Figure  2:  Notational  Roadmap 


3  The  SQAE  methodology  and  Framework  was  developed  by  and  can  be  licensed  through  the 
MITRE  Corporation. 

®  Octave  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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By  defining  the  characteristics  that  make  these  methods  suitable  for  inclusion  in  the  RDR,  it 
would  then  be  possible  to  identify  others  like  them.  The  framers  of  the  roadmap  started  with 
the  methods  most  familiar.  If  an  explanation  of  the  characteristics  that  qualified  them  for  the 
list  could  be  defined,  other  diagnostics  could  be  added  and  analyzed.  The  following 
characteristics  quickly  emerged  as  the  key  qualifiers: 

•  Risk  identification  phase 

•  Analysis  phase 

•  Potential  risk  statement  “leave-behind” 

We  emphasize  that  this  is  a  notional  roadmap.  The  team  has  given  only  the  most  preliminary 
thought  to  the  connections  between  the  methods  shown.  The  colors  of  these  connections  are 
likewise  only  suggestive,  and  will  probably  gain  meaning  and  utility  as  we  expand  and  refine 
the  RDR. 

The  hexagons  with  solid  line  borders  represent  the  methods  we  are  already  studying  or 
considering  for  inclusion.  The  hexagons  with  dotted-line  borders  represent  areas  that  we 
think  should  contain  a  method,  but  where  team  members  did  not  know  of  any  risk-based 
diagnostics  that  fulfilled  the  identified  function.  In  these  areas,  collaborators  could  identify 
existing  diagnostics  that  fulfill  the  identified  functions  or  choose  to  do  development  in  that 
area  to  fill  the  gap  in  the  Risk  Diagnostics  Roadmap. 

The  “cloud”  at  the  bottom  of  the  drawing  represents  a  class  of  methods  that  will  be  necessary 
for  the  foreseeable  future  until  more  structured  diagnostics  can  be  developed.  These  are  the 
expert-based  inquiries — so  called  “red  teams”  or  “graybeard  panels”  that  include  perhaps 
most  of  the  independent  technical  assessments  that  the  SEI  has  conducted  over  the  years. 
They,  too,  have  a  place  on  the  RDR,  and  their  results  should  also  be  entered  into  the  risk 
repository  as  structured  risk  items. 
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5  Next  Steps 


Since  the  team  investigating  risk  diagnostics  initially  focused  initial  on  a  small  sampling  of 
diagnostics,  an  obvious  next  step  is  to  include  techniques  from  outside  the  SEI.  Early 
candidates  include  the  SQAE  developed  by  MITRE,  and  the  OCTAVE  approach  developed 
by  the  SEI.  Once  the  RDR  is  populated,  it  will  be  possible  to  construct  both  retrospective 
and  prospective  reviews,  ensuring  that  the  roadmap  becomes  a  comprehensive  reference  of 
all  available  RI&A  techniques.  Another  long  term  goal  of  this  effort  will  be  a)  developing 
standardized  descriptions  for  risk  items  identified  by  the  various  diagnostic  techniques  and  b) 
ensuring  that  those  standardized  risk  items  are  referenced  in  the  RDR. 

For  the  Risk  Diagnostics  Roadmap  to  be  useful  and  relevant  in  any  industry  area,  it  must  be 
continually  tested  and  revised  based  on  the  current  capabilities,  the  latest  technology  trends, 
and  the  pressing  business  drivers.  A  critical  mass  of  professionals  in  risk,  perhaps  a  panel  of 
both  practitioners  and  consumers,  must  challenge  the  roadmap’s  assumptions  and 
conclusions.  Ideally,  some  body  of  knowledgeable  professionals  should  review  and  update 
the  Risk  Diagnostic  Roadmap  at  regular  intervals. 

We  would  like  other  stakeholders  in  the  acquisition  community  to  share  our  sense  that  the 
RDR  is  a  synergizing  vision  that  will  benefit  us  all — acquisition  programs,  the  organizations 
that  serve  them  (e.g.,  federally  funded  research  and  development  institutions,  consultants), 
and  developers. 

We  are  looking  for  potential  collaborators  and  funding  to  help  us  identify  additional  risk 
identification  and  analysis  techniques  for  possible  inclusion  in  the  roadmap. 
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